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@ Program Identification system. 

@ A system for identifying programs such as television 
programs received from various sources detects the occur- 
rence of predetermined events such as scene changes in a 
video signal and extracts a signature from the video signal. 
The signatures and the times of occurrence of the sig- 
natures are stored and subsequently compared with re- 
ference signatures to identify the program. The signatures 
may be compared in pairs to Increase resolution, and the 
time interval between events or signatures may also be 

I" used to Identify the program, either by themselves or In 

£ conjunction with the signatures. 
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Program Identification System 

* 

This invention relates generally to methods 
and apparatus for identifying programs and the viewing 
habits of the public, and more particularly to a 
system for identifying programs, such as tele- 

vision programs obtained from various sources, includ- . 
ing video recorders. 

Systems for identifying programs such as 
television programs that are broadcast or viewed are 
known. Such systems fall into various categories. 
These categories include manual systems wherein the 
viewer keeps a diary indicating all of the programs 
viewed; mechanical, electromechanical and electronic 
systems that sense the channel to which a television 
receiver is tuned; systems that detect identifying 
signals present in the television broadcast signal; 
and systems that monitor and analyze the program con- 
tent of the broadcast signal utilizing image proces- 
sing and correlation techniques to identify the pro- 
gram. 

While all of these systems do provide a way 
to identify programs and to monitor the viewing habits 
of the public, the manual methods are slow in acquiring 
data and are prone to inaccuracies resulting from the 
entry of erroneous data that may be intentionally or 
unintentionally entered. Systems that monitor the 
channel to which a receiver is tuned overcome the 
drawbacks of the manual systems, but require access 
to the tuning mechanism of the receiver. Such access 
is becoming increasingly difficult with the advent of 
cable and pay television systems that utilize various 
converters and decoders, many of which are difficult 
to access. Moreover, such systems cannot identify 
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programs played from a source such as a home video 
recorder. Systems that detect identification signals 
encoded on the broadcast signal overcome many of the 
accuracy and access disadvantages described above , 
but require the cooperation of the broadcasters who 
must encode the programs prior to broadcast for such 
systems to be effective. Consequently, such systems 
are useable only on a limited number of broadcasts, 
such as network broadcasts. Systems that monitor and 
analyze the content of the program itself theoretical- 
ly eliminate the disadvantages of the other systems, 
but heretofore the information processing requirements 
of such systems rendered such systems impractical or 
uneconomical because of the computing capacity requir- 
ed to analyze the vast quantity of information present 
in a broadcast signals. Moreover, previous attempts 
to reduce the quantity of information processed tend- 
ed to introduce inaccuracies. 

An example of a system that utilizes the 
program content of a broadcast signal to identify the 
program is described in United States Patent No. 
3,919,479 to Moon et al. The Moon et al. system 
utilizes a non-linear analog transform to produce a 
low frequency envelope waveform, and the information 
in the low frequency envelope of a predetermined time 
interval is digitized to generate a signature. The 
signatures thus generated are compared with reference 
signatures to identify the program. However, the 
Moon et al. system generates the signatures continuous- 
ly, and -consequently, requires a large computer to 
process the data thus generated. 

Another system that monitors the program 
content of a broadcast signal is disclosed in United 
States Patent No. 4,230,990 to Lert Jr. et al. The 
system disclosed in the Lert Jr. et al patent reduces 
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the amount of data that must be processed, as compared 
to the data amount as processed my the Moon et al. patent, 
by utilizing cues that are either externally generated 
or present in the program material to initiate the signa- 
ture generation and correlation process only after a cue 
has been detected. However, the system disclosed in the 
Lert Jr. et al. patent is designed to monitor the pro- 
grams broadcast by broadcasting stations such as network 
affiliated stations in order to determine whether they 
are broadcasting the programs required by the networks 
to be broadcast, rather than as a system for determining 
the viewing habits of the public. Thus, many of the prob- 
lems that occur when the habits of a television viewer 
are being monitored, such as, for example, frequent channel 
changing, and the problems that occur in identifying pro- 
grams that are recorded and subsequently played back by 
a home video recorder are not addressed. 

The invention as claimed solves the problem of identify- 
ing programs broadcast and the viewing habits of televi- 
sion viewers regardless of the source of the program 
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material, whether it be off the air, cable or prerecorded 
material, utilizing data reduction and correlation tech~ 
niques to minimize the amount of computation required 
to identify the program material. This is accomplished 
by extracting signatures from the program material only 
upon the occurrence of certain predetermined events, 
whether the events occur in the program material, for 
example a blank frame or a scene change, or are the result 
of an action taken by the viewer, for example the on/off 
condition of the television set and the mode of operation 
of the video recorder. In addition to providing a trigger 
for the extraction of the signature from the program 
material, the predetermined sequence of events may serve to 
indicate whether a program is being recorded or a prere- 
corded program is being viewed. 

In accordance with a preferred embodiment of the invention, 
there is provided a system having a home unit that monitors 
the viewing conditions in the home, for example, whether 
the television set is on or off, whether any home video re- 
corder (e.g., a video cassette recorder or VCR) is on or off 
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and the video signal being received. By monitoring 
the aforementioned signals and functions, the system 
determines whether no viewing is occurring, whether a 
television broadcast is being viewed, whether a tele- 
vision broadcast is being recorded, whether viewing 
and recording is occurring simultaneously, or whether 
a previously recorded program is being viewed. Once 
the viewing mode has been established, the program 
being viewed is identified by extracting a character- 
istic signature from the video signal and storing it 
for subsequent comparison with reference signatures 
stored in a central office. 

Once the mode of viewing has been determin- 
ed, a signature of the program material being viewed 
or recorded is extracted, provided certain predeter- 
mined events occur first. These events shall be re- 
ferred to as Event 1 and Event 2. Several events can 
serve as Event 1. These events may be a turn on of 
the television set, a channel change, a predetermined 
time interval, for example 5 to 10 minutes, since the 
occurrence of the previous Event 1, or a sudden 
change to a black scene. Events that can serve as an 
Event 2 include a scene change to a scene other than 
a black scene, a color change, which may be an overall 
color change or a color change of a predetermined 
line, a still picture as evidenced by several 
consecutive substantially identical frames, an audio 
level change, and others. 

Upon the occurrence of an Event 1, the home 
unit monitors the broadcast for the occurrence of an 
Event 2, and extracts a pair of signatures from the 
video signal immediately following the next two Event 
2s following an Event 1. Or, if a previously 
recorded program is being viewed, the system extracts 
three or more signatures from the video signal 
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immediately following the next three or more Event 2s 
following an Event 1. The signatures are extracted 
by detecting and digitizing the envelope of the video 
signal immediately following the Event 2s. The digi- 
5 tixed signatures, as well as the times that the 
signatures were extracted, are stored for future 
comparison with reference signals stored in a central 
location. The time intervals between Event 2s are 
also stored and used to identify the program either 

10 by themselves or in conjunction with extracted 

signatures* The time intervals between Event 2s are 
particularly useful for identifying the playback of 
previously recorded video tapes for reasons that will 
be discussed in a subsequent portion of the 

15 specification. 

The unit used to acquire the reference sig- 
natures that are compared with the signatures 
received from the home units is similar to the home 
unit except that the reference signature acquisition 

20 unit monitors each broadcast continuously, and does 
not require an Event 1 to initiate a signature 
extraction. Therefore, each broadcast is 
continuously monitored and a signature is extracted 
each time an Event 2 is detected. The signatures, as 

25 well as the times of occurrence of the signatures are 
stored to form a library of reference signatures. 
The stored signatures are compared with signatures 
obtained from the home unit that occurred at 
approximately the same time. The time intervals 

30 between Event 2s are also stored for comparison with 
corresponding time intervals received from the home 
. units. 

A mode of implementation and the major advantages 
35 of the present invention will become readily apparent 
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upon consideration of the following detailed descrip- 
tion and attached drawing wherein: 

PIG. 1 is a block diagram of a home unit 
of the program identification system according to the 
5 present invention; 

FIG. 2 is a block diagram of a central 
office and reference signature extraction portions of 
the system according to the invention; 

PIG. 3 is a chart illustrating the various 
10 modes of viewing and recording that can occur in a 
typical household; 

PIGS. 4 and 5 are flow charts illustrating 
the logical steps performed by the home and central 
office units, respectively; 
15 fig. 6 is a flow chart illustrating the 

logical steps performed by the central office unit in 
identifying signatures; and 

FIG. 7 illustrates how signatures and scene 
changes may be extracted from the video envelope. 

20 

Referring now to the drawing, with 
particular attention to FIG • 1, there is illustrated 
a blocK diagram of the home unit of the system 
according to the invention generally designated by 

25 the reference numeral 10. The system 10 receives 

signals to be identified from a television receiver 
12, and in many instances from a home video recorder, 
such as a video cassette recorder or a VCR 14. In a 
typical home installation, the receiver 12 and the 

30 VCR 14 receive signal from a receiving antenna 16, 

and in many instances from a cable television system 
18. These signals may be applied directly to the 
receiver 12 and to the VCR 14, but it is convenient 
to employ a switching network 20 that selectively 

35 switches the receiver 12 and VCR 14 to the antenna 16 
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and to the cable 18 in order to permit the viewing 
and recording of broadcasts received from the antenna 
16 or from the cable 18. In addition, the switching 
network 20 may connect the VCR 14 to the receiver 12 
5 to permit prerecorded or previously recorded tapes to 
be played bacK by the VCR and viewed by the receiver 
12. 

Video signals representative of the program 
applied to the receiver 12, as well as video signals 

10 representative of the signals being recorded by the 
VCR 14, are applied to a video processing circuit 22, 
for example, by a video switching circuit 24 which 
permits either the receiver 12 or the VCR 14, or both, 
to be monitored. The video processing circuit 22 

15 includes a detector and a low pass filter (discussed 

subsequently in greater detail), and provides a signal 
representative of the envelope of the video signal to 
an analog-to-digital converter 25 which generates a 
digital representation of the video envelope. The 

20 digitized video envelope signal is applied to an events 
detector 26 which detects predetermined events that 
occur in the digitized video signal, and causes the 
signature of the digitized video signal to be extract- 
ed by signature extraction circuitry 28 upon the oc- 

25 currence of a predetermined sequence of events. The 
signature thus generated is stored in a data storage 
system 30 along with the time that the signature was 
extracted. The extraction times are provided by a 
clock 32 which provides a digital representation of 

30 time to the data storage system 30 via, for example, 
a combining circuit 34. The clock 32 may be a real 
time clock or a relative time clock that is periodical- 
ly reset from a central location. The function of 
the events detector 26, the signature extractor cir- 

35 cuitry 28, the data storage 30 and the clock 32 can 
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be performed by individual circuits as shown or by a 
microprocessor based system. 

The home unit 10 is interrogated at periodic 
intervals, for example, once a day, by a central of- 
5 fice unit, generally designated by the reference numer- 
al 40 (PIG. 2) , which compares the signatures from 
the various home units with reference signatures in 
order to identify the signatures from the home unit. 
The home units may be periodically interrogated by a 

10 data communications circuit 42 that accesses the vari- 
ous home units via a suitable communications system, 
for example, a plurality of telephone lines 44. The 
signatures thus collected are compared by a central 
computer system 48 with signatures stored in a data 

15 base 46. Central computer system 48 controls the 

collection and classification of the signatures re- 
ceived from the home unit as well as the generation 
of reference signatures to be stored in the data base 
46. The latter function is performed in conjunction 

20 with a plurality of reference signal extraction cir- 
cuits 50 that are located in the cities being monitor- 
ed. The reference signature extraction circuits moni- 
tor the various networks, cables and other signal 
sources in those cities, and extract reference signals, 

25 for example, whenever an Event 2 occurs in any of the 
signals being monitored. The reference signals thus 
extracted are transmitted to the central office unit 
40 and stored in the data base 46 along with the times 
that such signatures are extracted, as provided by a 

30 clock 52. The clock 52 of PIG. 2 is similar to the 
clock 32 of the home unit and serves to indicate the 
time of occurrence of the reference signatures. The 
clock 52 may also be a real time clock or a relative 
time clock that is periodically set by the central 

35 office 40. A central reference signature extraction 
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circuit 54 and associated clock 56 serve to extract 
signatures and times of extraction of cable originated 
programs and signatures of prerecorded programs. Al- 
ternatively, extraction circuits may be placed at the 
5 head ends of cable system to extract reference signa- 
tures of cable programs. The signature extraction 
performed by the various extraction circuits and data 
storage is controlled by the central computer system 
48. 

10 The home unit of the system according to 

the invention monitors the mode of viewing by monitor- 
ing the on and off and other functions of the televi- 
sion receiver and any video recorder that may be used 
in the home. The various modes of viewing or recording 

15 are illustrated in FIG. 3. Mode 1 occurs when both 
the VCR and television receiver are both off, and 
indicates that no viewing is taking place. Mode 2 
represents television viewing and occurs when the 
television receiver is on and the VCR is off. Mode 3 

20 represents recording by the home VCR and occurs when 
the television receiver is off and the VCR is on. 
Mode 4 occurs when the VCR and the television receiver 
are both on, and can represent one of three conditions. 
The first condition, designated as mode 4.0, represents 

25 television viewing through the tuner of the VCR with- 
out recording. Mode 4.1 represents VCR recording and 1 
television viewing* while the third condition, desig- 
nated as Mode 4.2, represents VCR playback. Other 
modes of operation of the VCR, such as fast forward 

30 and rewind, are designated as Modes 4.3 and 4.4, 

respectively. Most of these modes can be easily de- 
tected by monitoring power line voltage or voltages 
elsewhere in the television set and the video recorder. 
Modes 4.0, 4.3 and 4.4 can be detected by logic cir- 

35 cuitry. 
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As previously stated, the home unit 10 moni- 
tors the viewing mode and the video signal, and ex- 
tracts signatures from the video signal only if cer- 
tain events occur. The logic necessary to provide 
this function is illustrated in PIG. 4. As is appar- 
ent from FIG. 4, the mode of operation is recorded 
whenever there is a change in the mode, such as, for 
example, a change from viewing to recording, etc. 
Each time the mode changes, a determination is made 
in order to determine what the new mode is. This is 
accomplished by determining whether the television is 
on or off, whether the VCR is on or off, and whether 
the VCR is in a record or playback mode. Once it has 
been determined whether the television and VCR are on 
or off, and whether the VCR is in a record or playback 
mode, the mode can readily be determined by using a 
look-up table containing the information in PIG. 3, 
and the mode is recorded. 

If the mode is determined to be Mode 1, 
indicating no viewing, no further action is taken 
until the mode changes again. If it is determined 
that the mode of operation is either Mode 2 or Mode 
4, the system monitors the television receiver for 
the occurrence of an Event 1, which may, for example, 
be any one of the following: 

1. television turn-on 

2. a channel change 

3. a predetermined time interval (e.g 5- 
10 minutes) since the previous Event 1 

4. a sudden scene change to a black scene. 

5. loss of synchronization for an appreci- 
able period of time. 

Such Event Is are relatively easy to detect. For 
example, television turn on can be detected simply by 
monitoring an appropriate .voltage or current. Televi- 
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sion turn-on is not used as an Event 1 in the system 
illustrated in PIG. 4 but may be used in other systems, 
particularly systems that do not monitor a VCR and 
consequently need not employ mode logic. A channel 
change can be detected by monitoring the position of 
the tuner mechanism when mechanical tuners are used, 
by monitoring tuning voltage in electrically tuned 
tuners v by monitoring local oscillator frequency, or 
even more simply by monitoring the video synchroni- 
zation pulses, preferably the vertical syncronization 
pulses and indicating a channel change in the event 
of a loss or a change in the syncronization pulse for 
a short time interval. The predetermined time interval 
can readily be determined from a clock, and a sudden 
scene change to black can be readily detected by moni- 
toring the average value of the video signal and indi- 
cating a change to a black scene when the average 
value becomes that representative of an all black 
scene. A loss of synchronization for an appreciable 
length of time (longer than that caused by a channel 
change) indicates fast forward or rewind mode of opera- 
tion of a VCR (Mode 4.3 or 4.4) or the loss of the 
video signal (e.g. station off the air). 

As long as no Event 1 occurs, the home unit 
records no new data; however, upon the occurrence of 
an Event 1, the system is conditioned to extract a 
signature upon the occurrence of the next two consecu- 
tive Event 2s, with an Event 2 being defined as: 

1. a scene change to a scene other than a 
black scene 

2. a color change in successive frames or 
portions thereof 

3. a still picture 

4. an audio level change. 
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Ways of detecting the various Event 2s described above, 
as well as ways for extracting signatures will be 
discussed in a subsequent portion of the specification. 

Upon the occurrence of an Event 1, a counter 
5 is set to zero and the video signal is monitored to 
determine if the television synchronization or sync 
signal is stable. This is accomplished, for example, 
by monitoring the vertical synchronization pulses and 
waiting until a predetermined number, for example, 

10 16, consecutive stable vertical sync pulses are detect- 
ed before the unit is permitted to detect an Event 2. 
If the synchronization has not been stable for a signi- 
ficant period of time, more than five seconds, this 
period will be marked either as television station is 

15 off the air or VCR fast forward or rewind. 

Once the sync has been stable for the requir- 
ed number of pulses, the system monitors the video 
signal for the occurrence of an Event 2. Whenever 
such an Event 2 change is detected, the time of the 

20 occurrence of the Event 2 is recorded as is the time 
interval since the previous Event 2, if the time in- 
tervals between Event 2s are to be used as a means of 
program identification. The signature of the first 
frame following the last detected Event 2 change is 

25 then extracted. Since a standard broadcast television 
frame consists of two interlaced fields, and since 
the two fields that form a frame are quite similar, 
it is not necessary to analyze both fields of the 
frame to obtain a signature. Typically, a satisfactory 

30 signature is obtained from only a single field, for 
example, the first field of a frame. The method of 
extracting the signature will be discussed in greater 
detail in a subsequent portion of the detailed descrip- 
tion, as will the method of detecting an Event 2. 
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Once the signature has been extracted, the 
counter is incremented by 1. if the count in the 
counter is less than two, indicating that more signa- 
tures are needed, the system is conditioned to respond 
to another Event 2 to extract and record another signa- 
ture, if the count in the counter is two (or the 
required number of signatures) or greater, indicating 
that the signatures required have been extracted, the 
system is conditioned to require an Event 1 to occur 
before it responds to any further Event 2s. 

If it is determined that the mode of opera- 
tion is Mode 3, that is, VCR recording, it is not 
necessary for an Event 1 to occur before the signal 
is monitored for Event 2s. The only determination 
that is made is whether the synchronization is stable, 
and if so, the system looKs for Event 2s continuously 
and extracts the signature and records the time of 
occurrence of the signature each time an Event 2 is 
detected. Thus, all signatures following every Event 
2 are extracted and stored so that upon subsequent 
playback, the program may be identified even though 
only portions of the tape may be played bacfc. 

The logic for extracting reference signa- 
tures is illustrated in FIG. 5. Each broadcast and 
cable channel is monitored continuously by the refer- 
ence signature extraction circuits 50 and 54, and a 
signature is extracted each time an Event 2 occurs to 
provide a reference library of reference signals. 
Consequently, since it is desired to extract a signa- 
ture that occurs after every Event 2, Event Is are 
not monitored. All that is required is that the sync 
be stable and that an Event 2 occur. Thus, whenever 
an Bvent 2 occurs, the time of occurrence of the Event 
2 as well as the signature of the scene following the 
scene change is recorded for future comparison with 
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signatures received from the various home units. The 
time intervals between Event 2s may be recorded or 
obtained from the times of occurrence of the Event 
2s. 

The logic employed by the central computer 
system 48 of the central unit 40 in order to identify 
a program is illustrated in PIG. 6. As is illustrated 
in PIG. 6, two consecutive signatures from a home 
unit are selected. If there is not enough data to 
define two consecutive signatures, the next home unit 
is monitored. After two consecutive signatures are 
selected, a check is made to determine whether the 
signatures really are a pair, for example, that there 
is no Event 1 between the two signatures. 

If the two signatures are a pair, the time 
associated with the first home unit signature is read. 
All reference unit signatures in the data base 46 
that occurred within a predetermined time interval of 
the occurrence of the first home unit signature, for 
example, plus or minus eight seconds, are selected 
for comparison, and the correlation coefficient be- 
tween the first home unit signature and the selected 
reference unit signatures are computed. If one of 
the correlation coefficient exceeds a predetermined 
threshhold, the time associated with the second home 
unit signature is read. The next several, for example, 
six, reference unit signatures that occur, for example, 
within a predetermined time interval, for example, 
plus or minus eight seconds of the time associated 
with the second home unit signature are selected, and 
the correlation coefficients between the second home 
unit signature and the reference unit signatures are 
computed. If one of the correlations coefficient 
exceeds the predetermined threshhold, an identifica- 
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tion is made and stored, and printed out, if neces- 
sary. 

As discussed above, the system according to 
the invention utilizes a time reference to access 
5 reference signatures that occurred at approximately 
the same time as the home unit signature being evalu- 
ated in order to eliminate the need for comparing the 
home unit signature with all of the reference signa- 
tures in the data base 46. However, when a previous- 
10 ly recorded program is being played back, the time 
that the signature occurs during playback cannot be 
used to locate the reference signal. Thus, in 
accordance with another important aspect of the 
present invention, the times of occurrence of signa- 
ls tures that occur when programs are being recorded are 
also stored* The times of occurrence of signatures 
that occur in Mode 3 (recording) are kept in a separ- 
ate recording time file in the data base 46. Similar- 
ly, in Mode 4.1 (viewing and recording) the times of 
20 signatures are also kept in the recording time file. 
Thus, when the broadcast is played back (Mode 4.2), 
even though the recorded signatures cannot be matched 
to signatures stored in the reference unit during 
playback, the signature can still be identified by 
25 utilizing the recording time instead of the playback 
time to retrieve the corresponding reference signa- 
tures from the reference signature data base 46. 
Thus, by keeping signatures relating to programs that 
were broadcast several days or weeks ago within the 
30 data base 46, such previously recorded and subsequent- 
ly played back broadcasts can be readily identified. 

Alternatively, in order to identify the 
playback of previously recorded programs, instead of 
comparing the obtained during playback with reference 
35 the signatures that represent the programs broadcast 
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several days or weeks ago stored in the recording 
time file of the reference signature data base 46 
(FIG. 2) of the central office unit, the played back 
signatures may be compared with signatures obtained 
5 by the home unit previously during VCR recording (Mode 
3) and stored in the data storage 30 (FIG. 1) of the 
home unit. If desired, the signatures may also be 
sent to the central office, as are the signatures 
obtained during normal television viewing, but these 

10 signatures would be labeled as VCR recording, and may 
be stored in the recording time file. However, when 
playback occurs, the played back signatures would be 
compared with the signatures stored in the home unit 
data storage. Only if a match occurred, would the 

15 signatures be sent to the central office and compared 
with the signatures stored in the central office data 
base 46 to identify the program. If no match occurred, 
the signatures would be sent. This is because what- 
ever is being played back cannot be identified, and 

20 may be something like, for example, a home movie. 
Thus, the system eliminates the need to search the 
entire recording file of the data base 46 in an attempt 
to identify something that cannot be identified. 

In the playback mode, when attempting to 

25 match the signatures obtained during playback with 
the signatures stored in the data storage 30, it is 
convenient to use the time intervals between the occur- 
rence of the signatures (or Event 2s) to generate a 
time interval signature, TIS, (discussed in greater 

30 detail in a subsequent portion of the specification) 

to obtain the match. When using a time interval signa- 
ture for matching, three or more consecutive signatures 
are extracted, and the time intervals between the 
signatures are compared with the time intervals between 

35 signatures stored in the data storage 30. When a 
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corresponding pair of consecutive time intervals is 
found, a match is indicated. If greater accuracy is 
desired, more than three consecutive signatures may 
be extracted to provide more than two consecutive 
5 time intervals. 

Event 2s may be detected in a variety of 
ways. For example, the video signal may be low pass 
filtered, for example, through a low pass filter hav- 
ing a cut-off frequency on the order of approximately 

10 2 to 6 kiloHertz, and its average amplitude detected. 
The signal may be digitized by assigning a number 
representative of the average amplitude of the video 
signal for each frame. The numbers thus obtained may 
be compared in various ways. For example, the numbers 

15 obtained from different frames may be compared, and 
if the numbers are substantially different, a scene 
change will be indicated. The numbers compared may 
be obtained from consecutive frames or from non-conse- 
cutive frames, for example, frames spaced by a 20 to 

20 40 frame interval, depending on the type of event 

that has been defined as the Event 2. Alternatively, 
a moving average of the numbers representative of the 
video amplitudes of the last several frames may be 
calculated, and the number representative of the aver- 

25 age amplitude of the last detected frame compared 

with the moving average to indicate a scene change if 
the number associated with the last frame substantial- 
ly differs from the moving average. 

The various comparing schemes are useful 

30 for identifying various different types of Event 2s. 
For example, comparing the number associated with a 
particular frame with the number associated with a 
previous frame is useful for detecting an Event 2 
that may be termed an instant scene change. A moving 

35 average comparison or a comparison with frames that 
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occurred several frames ago is useful for detecting 
an Event 2 that may be termed a fade type scene change 
wherein one scene is gradually faded in and the pre- 
vious scene is gradually faded out. Because in a 
5 fade type scene change, the change occurs gradually, 
there is no drastic change in the number associated 
with consecutive frames , and consequently, such a 
scene change cannot readily be detected by a system 
that compares immediately successive frames. Numbers 

10 that are substantially identical for several consecu- 
tive frames may be used to indicated a still picture. 

Other methods of detecting an Event 2 in- 
clude detecting a color change, for example, by de- 
tecting the average amplitude of a particular color 

15 component, such as, for example, a red component, of 

each frame or a particular line or lines of each frame, 
and making comparisons of the types described above. 
Alternatively, different color components or an aver- 
age of the three color signals can be monitored for 

20 each frame and comparisons made. In addition, rather 
than monitoring an entire frame, only portions of a 
frame may be monitored. These portions may be select- 
ed to be those portions of a frame that normally do 
not change unless a scene change occurs. Such portions 

25 may occur near the top or bottom of the screen because 
such areas are less likely to be affected by action 
or movement occurring within a scene. The Event 2s 
may be detected electronically by monitoring the video 
envelope, or by monitoring the cathode ray tube, for 

30 example, by placing a light sensitive photoelectric 
device 36 (FIG. 1) in proximity with the cathode ray 
tube, preferably in one corner thereof. Alternatively, 
the audio signal may be monitored by an audio proces- 
sing circuit 38 and applied to the analog-to-digital 
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converter 25 to generate an Event 2 when the audio 
signal departs from a predetermined level. 

In accordance with an alternative embodiment 
of the present invention that is particularly suitable 
5 for identifying prerecorded programs, the time inter- 
vals between events, or in changes in one or more 
predetermined parameters, can be used to define a 
feature string. Such a system can be used either 
alone, or in combination with a signature extracting 

10 system to identify the program. When used alone, the 
amount of data generated by the system is less than 
that generated by a signature generating system, and 
therefore, does not require much computation time to 
obtain a correlation. Consequently, it is not be 

15 necessary to store the time of occurrence of each 

change (unless it is desired for other reasons) , but 
rather, only the elapsed time between such changes 
need be stored. Such a system is particularly useful 
in identifying the playback of prerecorded programs, 

20 or delayed broadcast programs because real time data 
is not necessary to make the identification. 

In a system wherein the intervals between 
predetermined changes in predetermined events are 
used to generate a time interval signature (TIS) • 

25 The time interval signature, which can be represented 
as a digital code sequence, can be constructed by 
measuring the time intervals between the sequential 
occurrences of regularly occurring events in a broad- 
cast. Preferably, the chosen events should have the 

30 following properties: 

1. Be readily measurable automatically. 

2. Occur at a sufficiently high rate that 
enough intervals are available to allow unique identi- 
fication of the signal. 

35 3. Always be present 
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4. Cannot be controlled by program origi- 
nators to render identification vulnerable to changes 
in programming methods. 

Typical events that can be used as the events 
5 to generate a time interval string includes scene 
changes to black, Event 2s , including shifts in the 
average amplitude of the video signal, departures in 
the amplitude of the instantaneous video signal from 
the average, shifts in the video color balance, changes 

10 in the average color of a predetermined line, for 

example, a line near the top of the screen, a predeter- 
mined audio power level, for example, silence, etc. 
Thus, to define a time interval string, the home unit 
would a generate time interval string (TIS) by detect- 

15 ing one of the predetermined events, for example, an 
Event 2 such as a scene change, and store the time 
intervals between consecutive changes. 

The monitoring of events can be done continu- 
ously to generate a long TIS without requiring an 

20 excessive memory capacity because such changes occur 

relatively infrequently, and the time intervals there- 
between can be defined by only a few bits. The TIS's 
thus obtained from the home unit are compared with 
reference TIS's stored in the central unit. 

25 However, in order further to reduce the 

amount of data that must be stored, the time intervals 
between the events being monitored may be monitored 
only after the occurrence of another predetermined 
event, such as an Event 1; and it has been found that 

30 satisfactory identification can be obtained by gener- 
ating a feature string consisting of three time inter- 
vals between changes in the monitored variable, such 
as a scene change or other Event 2. The time interval 
strings thus generated are matched with reference 

35 time interval strings that are obtained in much the 
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same way as those generated by the home units except 
that the reference time interval strings are continu- 
ously generated for each broadcast or other program 
source being monitored. 

Referring now to Fig. 7, there is shown a 
video envelope 100 as provided from the video proces- 
sing circuit 22 (Pig. 1) to the analog-to-digital 
converter 25. The analog-to-digital converter 25 
takes a plurality of samples 102 (Fig. 7) for example, 
80 samples during a single frame interval, and gener- 
ates a digital value representative of the amplitude 
of each of the samples 102. The digital values thus 
generated are applied to the events detector 26 (Fig. 
1) and to the signature extraction circuit 28 and 
used to detect the various events and to extract sig- 
natures. 

in order to detect an Event 2, for example, 
a scene change, the events detector 26 compares each 
of the digitized samples 102 of a current frame with 
a corresponding digitized sample 102 of a previous 
frame, either the immediately preceding frame or 
another previous frame, depending on the type of 
scene change to be detected. The counterpart samples 
are then compared to determine whether a scene change 
has occurred. More specifically, the detection of a 
scene change occurs as follows: 

1. Take, for example, 80 samples of the 
video envelope of each frame 

2. Compare counterpart samples of a cur- 
rent frame and a previous frame 

3. Take the absolute value of the differ- 
ence of each of the counterpart samples 

4. Take the average of the absolute values 

5. Indicate a scene change if the average 
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of the absolute values exceeds a pre- 
determined threshhold. 
The indication of a scene change is applied to the 
. signature extraction circuit 28 (Pig. 1) to cause the 
5 signature extraction circuit 28 to operate on the 
digitized samples 102 to extract the signature. 

The signature extraction circuit 28 operates 
on the digitized samples 102 by talcing the average 
value of consecutive samples 10 2 f for example, eight 
10 consecutive samples to generate a total of, for example, 
ten features 104 that comprise an image signature. 
Typically, four bits define a feature, and consequent- 
ly, an image signature can be defined by forty bits 
of data for the example discussed above? however, in 
15 some instances more features and more data may be 
required. 

Both the home unit 10 (Pig. 1) and the refer- 
ence signature extraction circuits 50 and 54 (Pig. 2) 
operate in the same manner to extract signatures, the 

20 only difference in operation being that the signature 
extraction circuit 28 (Fig. 1) requires an Event 2 to 
be preceded by an Event 1 before signature extraction 
occurs? but the reference signature extraction cir- 
cuits 50 and 54 (Fig. 2) do not require an Event 1 to 

25 precede an Event 2, but rather extract a reference 
signature each time an Event 2 occurs. 

Once the home unit signatures and the refer- 
ence signatures are obtained, the home unit signatures 
will be compared with the reference unit signatures, 

30 normally with those signatures occurring in correspond- 
ing time intervals. The correlation coefficients 
between the home unit signatures and the reference 
unit signatures will be compared, and the reference 
signature that has the best correlation to the home 

35 unit signature will be used to identify the program 
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being viewed, provided that the correlation coef- 
ficient exceeds a minimum threshhold. If it does 
not, no natch will be indicated. 

Obviously, modifications and variations of 
5 the present invention are possible in light of the 
above teachings. Thus, it is to be understood that, 
within the scope of the appended claims, the invention 
may be practiced otherwise than as specifically de- 
scribed above. 



10 



Claims: 
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1. A system for identifying program material 
that identifies programs by extracting predetermined 
characteristics from the program material and record- 
ing fot analysis the predetermined characteristics, 
characterized by: 

means (26) for detecting a first predeter- 
mined event in the program material; 

means (26) for detecting a second predeter- 
mined event in the program material; and 

means (28) for extracting at least one pre- 
determined characteristic of the program material 
after the occurrence of a predetermined number of 
said first and second events. 

2. The system recited in claim 1 wherein 

said program material is that of a television video signal and said 
second event detecting means includes means for de- 
tecting a scene change from one video scene to another 
video scene. 

3. A system as recited in claim 2 wherein 
said television signal is received by a television 
signal receiver , and wherein said means for detecting 
said first event includes means for detecting the 
turn-on of the television receiver. 

4. A system as recited in claim 2 wherein 
said television signal is received by a television 
signal receiver , and wherein said first event detect- 
ing means includes means for detecting a change in 
the channel to which the television signal receiver 
is tuned. 

5. A system as recited in claim 1 wherein 
said means for detecting said first event includes 
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means for detecting a change from a video scene to a 
black scene. 

6. A system as recited in claim 1 wherein 
said first event detecting means includes means for 
detecting the elapse of a predetermined length of 
time after the occurrence of the previous first event. 

7. A system as recited in claim 6 wherein 
said detecting means includes means for setting the 
predetermined length of time to approximately 5 to 10 
minutes. 

8. A system as recited in claim 1 wherein 
said extracting means includes means for extracting 
said predetermined characeristic upon the third occur- 
rence of the second predetermined event following the 
occurrence of the first event, and wherein the pre- 
determined characteristic is the time interval between 
the second event and the immediately preceding second 
event that was not followed by a first event. 

9. A system for identifying signals having 
means for operating on the signal to be identified in 
order to extract a feature string, means for comparing 
the feature string with various feature strings cor- 
responding to known signals and for identifying the 
signal when the feature string of the signal to be 
identified correlates with one of the feature strings 
corresponding to a known signal within a predetermined 
tolerance, wherein the feature string extracting means 
is characterized by: 

means (26) for monitoring predetermined 
changes in a predetermined parameter of the signal to 
be identified; 
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means (26, 32) for determining the time 
intervals between the predetermined changes in the 
predetermined parameter; and 

means (28, 32, 34, 30) for utilizing the 
time intervals between successive predetermined changes 
to generate the feature string. 

10. The system recited in claim 9 wherein 
the signal to be identified includes an audio signal, 
and wherein said feature string extracting means in- 
cludes means for monitoring the amplitude of the audio 
signal and determining the time intervals that the 
amplitude of the audio signal departs from a predeter- 
mined level of amplitude. 

11. The system recited in claim 10 wherein 
the predetermined level amplitude is a level that 
corresponds substantially to silence, and said time 
interval corresponds to the time interval between 
successive periods of silence in the audio signal. 

12. The system recited in claim 9 wherein 
the signal to be identified includes a video signal, 
and wherein said feature string extracting means in- 
cludes means for determining events ("Event 2s") in the video 
signal and determining the time intervals between such 
events to generate the feature string. 

13. The system recited in claim 12 wherein the 
event determining means includes means for monitoring 
the light emanating from the display on which the 

video signal is displayed and indicating a scene change 
when the amount of light emanating from the display 
changes by a predetermined amount. 



o 
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14. The system recited in claim 13 wherein 
the display light monitoring means includes means for 
monitoring the light with a photoelectric device, and 

indicating a respective event when the output of the photo- 
electric device changes by a predetermined amount, 

15. The system recited in claim 12 wherein the 
event determining means includes means for determin- 
ing the average amplitude of the video signal and 
indicating an event when the average amplitude of 

the video signal changes by a predetermined amount. 

16. The system recited in claim 12 wherein 
the event determining means includes means for com- 
paring the video envelopes of two frames. 

17. The system recited in claim 16 wherein 
the video envelope monitoring means includes means 
for comparing the envelopes of at least one field of 
two consecutive frames* 

18. The system recited in claim 16 wherein 
the video envelope monitoring means includes means 
for comparing the envelopes of at least one field of 
two non-consecutive frames. 

19. The system recited in claim 18 wherein 
the video envelope comparing means includes means for 
comparing the video envelopes of two frames spaced at 
approximately twenty to forty frame intervals. 

20. The system recited in claim 12 wherein 
the event determining means includes means for 
monitoring the video envelopes of two frames, means 
for obtaining a predetermined number of samples from 
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each envelope, means for comparing the magnitudes of 
counterpart samples to obtain a difference in coun- 
terpart samples, and means for talcing the average of 
the differences' and for indicating an event whenever 
the average exceeds a predetermined level. 

21. The system recited in claim 20 wherein 
the comparing and counterpart samples difference tak- 
ing means includes means for taking the differences 
of the absolute values of the counterpart samples. 

22. The system recited in claim 12 wherein 

the step of determining respective events includes means for 
determining the average level of the video signal, 
means for determining the amplitudes of predetermined 
portions of the video signal, means for comparing the 
amplitudes of the predetermined portions of the video 
signal with the average amplitude of the video signal 
and assigning one of a first and a second designation 
to each portion depending on whether the amplitude of 
the predetermined portion exceeds or is less than the 
average amplitude, means for utilizing the designations 
so obtained to obtain a frame code, means for compar- 
ing the frame code with the frame code of a previous 
frame and indicating a change if the current frame 
code differs from the previous frame code by a pre- 
determined amount. 

23. The system recited in claim 10 wherein 
said signal to be identified includes a video signal 
and said video signal includes a color signal, wherein 
the feature string extracting means includes means 

for determining the time interval between predetermin- 
ed changes in the color signals. 
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24. The system recited in claim 23 wherein 
the color signal change determining means includes 
means for comparing a function of the color signal 
with the average of the color signal and determining 
the time intervals when the function of the color 
signal exceeds the average of the color signal and 
the time intervals when the function of the color 
signal is less than the average of the color signal 
to extract the feature string. 

25. The system recited in claim 24 wherein 
the function of the color signal and the average of 
the color signal is determined for only a single line 
of the frame. 

26 The system recited in claim 25 wherein 
the single line is near the top of the image. 

27. The system recited in claim 9 wherein said 
predetermined changes are second predetermined events as 
recited in claim 1. 

28. A system for identifying programs view- 
ed on a television receiver, wherein the programs are 
obtained from a video recorder and from an external 
program source, characterized by: 

means responsive to the on/off condition of 
the television; 

means responsive to the on/off condition 
and the play/record condition of the video recorder; 

logic means responsive to the television 
condition responsive means and the video recorder 
condition responsive means for providing an indication 
if the program being viewed is being obtained from 
the video recorder or from the external program source; 
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means for identifying the program being 
viewed, said identifying means being responsive to 
said logic means and operative to identify said program 
according to two different criteria depending on whether 
the program being viewed is obtained from the external 
program source or form the video recorder. 

29. A system for determining the viewing 
habits of a household having a video recorder capable 
of recording programs obtained from an external program 
source and playing video programs, and a television 
receiver for viewing programs obtained from the exter- 
nal program source and from the video recorder, said 
system having a home unit and a central unit, charac- 
terized by: 

means responsive to predetermined operating 
conditions of said television receiver and said video 
recorder to provide signals representative of televi- 
sion viewing, recording, and playback; 

means (26) for detecting during viewing a 
first predetermined event in the program material; 

means (26) responsive to the detection of 
said first predetermined event for detecting during 
viewing a second predetermined event in the program 
material; 

means (28) responsive to the detection of 
said second predetermined event for extracting during 
viewing at least one predetermined characteristic of 
the program material; 

means (30) for storing at the home unit the 
predetermined characteristic of the viewed program 
material and the time that the predetermined charac- 
teristic is extracted; 
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means (26) foe continuously detecting during 
recording second predetermined events in the pro- 
gram material; 

means (26) responsive to the detection of 
said second predetermined events for extracting during 
recording at least one predetermined characteristic 
of the program material; 

means (30) for storing during recording the 
predetermined characteristics and the times that the 
predetermined characteristics are extracted; . 

means (48) for comparing the predetermined 
characteristics stored during viewing and recording 
and their times of occurrence with reference predeter- 
mined characteristics stored in the central unit that 
occurred at substantially the same time to identify 
the programs viewed and recorded; 

means (26) for detecting during playback 
the first predetermined event in the program material; 

means (26) responsive to the detection of 
said first predetermined event for detecting during 
playback the second predetermined events in the program 
material; 

means (28) responsive to the detection of 
said second predetermined event for extracting during 
playback at least one predetermined characteristic of 
the program material; 

means (30) for recording during playback 
the predetermined characteristic; and 

means (48) for comparing the predetermined 
characteristic recorded during playback with reference 
predetermined characteristics stored at the central 
unit at the time of recording to identify the program. 

30. A system for determining the viewing 
habits of a household having a video recorder and a 
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television receiver, said system having a home unit 
and a central unit, characterized by: 

means responsive to predetermined parameters 
of said television receiver and said video recorder 
to provide signals representative of television viewing, 
playback and recording; 

means (26) for detecting during viewing a 
first predetermined event in the program material; 

means (26) responsive to the detection of 
said first predetermined event for detecting during 
viewing a second predetermined event in the program 
material; 

means (28) responsive to the detection of 
said second predetermined event for extracting during 
playback at least one predetermined characteristic of 
the program material; 

means (30) at the home unit for storing 
during viewing the predetermined characteristic and 
the time that the predetermined characteristic is 
extracted; 

means (26) for continuously detecting during 
recording second predetermined events in the pro- 
gram material; 

means (26) responsive to the detection of 
said second predetermined events for extracting during 
recording at least one predetermined characteristic 
of the program material; 

means (30) at the home unit for storing 
during recording the predetermined characteristics 
and the times of occurrence thereof; 

means (26) for detecting during playback 
the second predetermined events in the program material; 

means (28) responsive to the detection of 
said second predetermined events for extracting during 
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playback at least one predetermined characteristic of 
the program material; 

means (26, 28, 32, 34) at the home unit for 
comparing the time intervals between the detected 
events during playback and the time intervals between 
the events stored during recording; and 

means (48) responsive to a match provided 
by the comparing means for comparing the extracted 
predetermined characteristics with reference charac- 
teristics stored in the central unit to identify the 
program. 
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